DeepSynth: Automata Synthesis for Automatic Task Segmentation in Deep Reinforcement Learning

This paper proposes DeepSynth, a method for effective training of deep Reinforcement Learning (RL) agents when the reward is sparse and non-Markovian, but at same time progress towards requires achieving an unknown sequence high-level objectives. Our employs novel algorithm synthesis compact automata to uncover this sequential structure automatically. We synthesise human-interpretable automaton from trace data collected by exploring environment. The state space environment then enriched with synthesised so that generation control policy RL guided discovered encoded in automaton. proposed approach able cope both high-dimensional, low-level features non-Markovian rewards. have evaluated DeepSynth's performance set experiments includes Atari game Montezuma's Revenge. Compared existing approaches, we obtain reduction two orders magnitude number iterations required synthesis, also significant improvement scalability.